PRETRAINED LANGUAGE MODEL

Deep Bidirectional Transformers for Relation Extraction without Supervision (D19-61)

Challenge:	Existing task-oriented dialogue systems lack ontology-aware pretraining methods for task-orientated dialogue.
Approach:	They propose an ontology-aware pretrained language model (OPAL) for end-to-end task-oriented dialogue (TOD) . they propose to pretrain on large-scale contextual text data to bridge the gap between the pretraining method and downstream tasks.
Outcome:	The proposed model achieves an exciting boost and obtains competitive performance even without any TOD data on CamRest676 and MultiWOZ benchmarks.

Copied to clipboard

Yannis Papanikolaou, Ian Roberts, Andrea Pierleoni

Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing (2021.eacl-demos)

Challenge:	Existing frameworks for relation extraction use distant supervision instead of annotated data.
Approach:	They propose a framework to deal with relation extraction tasks without supervision . they use syntactic parsing and pre-trained word embeddings to extract relations .
Outcome:	The proposed framework outperforms baselines on four biomedical datasets and achieves slightly worse results than the state-of-the-art in three out of four data sets.

Copied to clipboard

Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Veyseh, Thien Huu Nguyen

Towards Open-Domain Topic Classification (2022.naacl-demo)

Challenge:	Trankit is a lightweight, pre-trained toolkit for multilingual natural language processing.
Approach:	They propose a transformer-based toolkit for multilingual natural language processing that trains pipelines over 100 languages and 90 pretrained pipelines for 56 languages.
Outcome:	The proposed tool outperforms existing pipelines over sentence segmentation, part-of-speech tagging, morphological feature tabbing, and dependency parsing while maintaining competitive performance over tokenization, multi-word token expansion, and lemmatization over 90 Universal Dependencies treebanks.

Copied to clipboard

Hantian Ding, Jinrui Yang, Yuqian Deng, Hongming Zhang, Dan Roth

Information Retrieval

Challenge:	Existing supervised classification models are insensitive to class names, but are no longer effective in open-domain tasks where the taxonomy is unbounded.
Approach:	They propose a topic classification system that accepts user-defined taxonomy in real time . they train a pretrained language model on a new Wikipedia dataset and train it on Wikipedia .
Outcome:	The proposed system improves over existing zero-shot models and performs competitively with weakly-supervised models trained on in-domain data.

Improving Complex Knowledge Base Question Answering via Question-to-Action and Question-to-Question Alignment (2022.emnlp-main)

Copied to clipboard

Yechun Tang, Xiaoxia Cheng, Weiming Lu

Question Answering

Challenge:	ALCQA addresses the semantic and structural gap between natural language and action sequences . a priori, the semantics of the question and action are not well understood .
Approach:	They propose an alignment-enhanced complex question answering framework which aligns questions and actions into sequences.
Outcome:	The proposed framework outperforms state-of-the-art methods on a CQA and WQSP dataset.

Dynamic Topic Modeling by Clustering Embeddings from Pretrained Language Models: A Research Proposal (2022.aacl-srw)

Copied to clipboard

Anton Eklund, Mona Forsman, Frank Drewes

RankGen: Improving Text Generation with Large Ranking Models (2022.emnlp-main)

Challenge:	Neural Topic Models (NTMs) are topic models that are created with the help of a pretrained language model.
Approach:	They propose to do Neural Topic Modeling by Clustering document Embeddings (NTM-CE) with a pretrained language model to create dynamic topic models.
Outcome:	The proposed model can be evaluated theoretically and practically using quantitative measurements of coherence and human evaluation to evaluate the model.

Copied to clipboard

Kalpesh Krishna, Yapei Chang, John Wieting, Mohit Iyyer

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2021.eacl-main)

Challenge:	Modern language models assign high probabilities to output sequences that are repetitive, incoherent, or irrelevant to the prefix.
Approach:	They propose a 1.2B parameter encoder model for English that scores model generations given a prefix.
Outcome:	The proposed model outperforms decoding algorithms on automatic metrics and human evaluations with English writers.

Copied to clipboard

Timo Schick, Hinrich Schütze

A Controlled Reevaluation of Coreference Resolution Models (2024.lrec-main)

Challenge:	Existing approaches to learning from examples are limited due to the vast number of languages, domains and tasks.
Approach:	They propose a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task.
Outcome:	The proposed approach outperforms supervised training and strong semi-supervised approaches in low-resource settings by a large margin.

Copied to clipboard

Ian Porada, Xiyuan Zou, Jackie Chi Kit Cheung

Cross-domain Analysis on Japanese Legal Pretrained Language Models (2022.findings-aacl)

Challenge:	a pretrained language model is used in state-of-the-art coreference resolution models.
Approach:	They evaluate five coreference resolution models and control for language model used . they find that encoder-based CR models outperform decoder--based models in accuracy .
Outcome:	The encoder-based model outperforms the decoder--based models in accuracy and speed . older model generalizes the best to out-of-domain textual genres .

Copied to clipboard

Keisuke Miyazaki, Hiroaki Yamada, Takenobu Tokunaga

Flambé: A Customizable Framework for Machine Learning Experiments (P19-3)

Challenge:	Existing studies do not care the performance of domain-adapted PLMs for a generic domain.
Approach:	They propose to use pretraining strategies to build pretrained language models specialised in the legal domain to improve their performance.
Outcome:	The pretrained language models can learn domain-specific and general word meanings simultaneously and can distinguish them.

Copied to clipboard

Jeremy Wohlwend, Nicholas Matthews, Ivan Itzcovich

Challenge:	Flambé is a machine learning experimentation framework built to accelerate the entire research life cycle.
Approach:	They propose a framework that allows users to write custom code but include that code as a component in a larger system.
Outcome:	The proposed framework enables users to write custom code but include that code as a component in a larger system which is represented by a concise configuration file format.

AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization (2021.findings-acl)

Copied to clipboard

Xinsong Zhang, Pengshuai Li, Hang Li

Robust Integration of Contextual Information for Cross-Target Stance Detection (2023.starsem-1)

Challenge:	Pre-trained language models such as BERT have shown great power in natural language understanding . fine-grained tokenizations have advantages and disadvantages for learning of pre-tried models .
Approach:	They propose a pretrained language model based on both fine-grained and coarse-grain tokenizations . they propose to use both tokenization techniques to learn pre-trained models .
Outcome:	The proposed model outperforms BERT on benchmark datasets for Chinese and English . it can perform better with the same computational cost as BERT, the authors show .

Copied to clipboard

Tilman Beck, Andreas Waldis, Iryna Gurevych

Segmenting Numerical Substitution Ciphers (2022.emnlp-main)

Challenge:	Existing stance detection models do not take into account relevant contextual information which allows for inferring the stance correctly.
Approach:	They propose an approach to integrate contextual information as text into pretrained language models by prompting large language models.
Outcome:	The proposed approach outperforms baselines on a large and diverse stance detection benchmark in a cross-target setup, i.e. for targets unseen during training.

Copied to clipboard

Nada Aldarrab, Jonathan May

Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021.acl-long)

Challenge:	Existing methods for deciphering historical substitution ciphers are difficult to crack . cipheries that are not segmented are still difficult to deciphere .
Approach:	They propose automatic methods to segment historical substitution ciphers using BPE and unigram language models.
Outcome:	The proposed methods achieve an average segmentation error of 2% on 100 monoalphabetic ciphers and 27% on 3 real historical homophonic cipheries.

Copied to clipboard

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, James Henderson

Automatic Marketing Theme and Commodity Construction System for E-commerce (2023.emnlp-industry)

Challenge:	State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model.
Approach:	They propose a framework that can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks.
Outcome:	The proposed framework improves performance on the well-known GLUE benchmark while adding only 0.29% parameters per task.

Copied to clipboard

Zhiping Wang, Peng Lin, Hainan Zhang, Hongshen Chen, Tianhao Li, Zhuoye Ding, Sulong Xu, Jinghe Hu

Fire Burns, Sword Cuts: Commonsense Inductive Bias for Exploration in Text-based Games (2022.acl-short)

Challenge:	Existing recommendation system invites experts to write marketing themes and select relevant commodities, which suffer from difficulty in mass production, poor timeliness and low online indicators.
Approach:	They propose to use pretrained language model to generate marketing themes and commodity consistency module to select relevant commodities for the generative theme.
Outcome:	The proposed system can generate popular marketing themes and select relevant commodities automatically and improve theme online effectiveness compared with state-of-the-art methods.

Copied to clipboard

Dongwon Ryu, Ehsan Shareghi, Meng Fang, Yunqiu Xu, Shirui Pan, Reza Haf

Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gaps (2022.aacl-main)

Challenge:	Existing RL agents are far away from solving text-based games due to their combinatorially large action spaces that hinders efficient exploration.
Approach:	They propose an exploration technique that injects external commonsense knowledge, via a pretrained language model, into the agent during training when the agent is the most uncertain about its next action.
Outcome:	The proposed method exhibits improvement on the collected game scores during the training in four out of nine games from Jericho.

Copied to clipboard

Hiroki Iida, Naoaki Okazaki

Context Generation Improves Open Domain Question Answering (2023.findings-eacl)

Challenge:	IR models with a pretrained language model outperform lexical approaches like BM25 for vocabulary mismatch.
Approach:	They propose an unsupervised domain adaptation method by filling vocabulary gaps by expanding queries and documents through an MLM.
Outcome:	The proposed method outperforms the current state-of-the-art domain adaptation method on datasets with a large vocabulary gap from a source domain.

Copied to clipboard

Dan Su, Mostofa Patwary, Shrimai Prabhumoye, Peng Xu, Ryan Prenger, Mohammad Shoeybi, Pascale Fung, Anima Anandkumar, Bryan Catanzaro

LangNav: Language as a Perceptual Representation for Navigation (2024.findings-naacl)

Challenge:	Existing closed-book question answering methods do not fully exploit the parameterized knowledge.
Approach:	They propose a closed-book QA framework which uses a coarse-to-fine approach to extract the relevant knowledge and answer a question.
Outcome:	The proposed method outperforms open-book QA methods on three QA benchmarks.

Copied to clipboard

Bowen Pan, Rameswar Panda, SouYoung Jin, Rogerio Feris, Aude Oliva, Phillip Isola, Yoon Kim

RadLing: Towards Efficient Radiology Report Understanding (2023.acl-industry)

Challenge:	Existing approaches to vision-and-language navigation use visual features as the perceptual representation of a visual representation of an agent's egocentric panoramic view.
Approach:	They propose to use off-the-shelf vision systems to convert an agent’s egocentric panoramic view into natural language descriptions.
Outcome:	The proposed approach improves on the R2R VLN benchmark by using synthetic trajectories from a prompted language model and domain transfer where a policy learned on one simulated environment (ALFRED) is transferred to another (more realistic) environment and combining both vision- and language-based representations.

Copied to clipboard

Rikhiya Ghosh, Oladimeji Farri, Sanjeev Kumar Karn, Manuela Danu, Ramya Vunikili, Larisa Micu

DAPPER: Learning Domain-Adapted Persona Representation Using Pretrained BERT and External Memory (2020.aacl-main)

Challenge:	a few pre-trained language models have produced comparable results in fine-tuning tasks in radiology domain.
Approach:	They propose a continuously pretrained language model with ELECTRA-small architecture that can compete with state-of-the-art results in radiology domain.
Outcome:	The proposed model can compete with state-of-the-art models for fine tuning tasks in radiology domain.

Copied to clipboard

Prashanth Vijayaraghavan, Eric Chu, Deb Roy

SEQˆ3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression (N19-1)

Challenge:	Empirical evidence suggests that the learnt persona embeddings can be effective in downstream tasks like hate speech detection.
Approach:	They propose a model that embeds personas from natural language into text . they evaluate the transferability of the model by simulating low-resource scenarios .
Outcome:	The proposed model can learn to embed persona from natural language and alleviate task or domain-specific data sparsity issues related to personas.

Copied to clipboard

Christos Baziotis, Ion Androutsopoulos, Ioannis Konstas, Alexandros Potamianos

Improved Near-Duplicate Detection for Aggregated and Paywalled News-Feeds (2025.naacl-industry)

Challenge:	Neural sequence-to-sequence models are currently the dominant approach in natural language processing tasks, but require massive parallel corpora.
Approach:	They propose a sequence-to-sequence-tosequnce autoencoder with words as latent variables . they apply the model to unsupervised abstractive sentence compression .
Outcome:	The proposed model achieves promising results in unsupervised sentence compression on benchmark datasets.

Copied to clipboard

Siddharth Tumre, Sangameshwar Patil, Alok Kumar

Pretrained Language Models for Dialogue Generation with Multiple Input Sources (2020.findings-emnlp)

Challenge:	News aggregators provide comprehensive and timely news stories that are sourced from diverse sources but differ in phrasing, formatting or supplemented with additional details.
Approach:	They propose a method that combines embeddings from pretrained language model and latent metadata of a news article followed by community detection to identify clusters of near-duplicates.
Outcome:	The proposed approach can detect nuanced similarities and differences in news snippets using pretrained language model and latent metadata of a news article followed by community detection.

Copied to clipboard

Yu Cao, Wei Bi, Meng Fang, Dacheng Tao

Dialogue Systems & Conversational Agents

Challenge:	Large-scale pretrained language models have achieved outstanding performance on natural language understanding tasks.
Approach:	They propose to fuse attention information from multiple input sources to achieve better relevance with dialogue history than simple fusion baselines.
Outcome:	The proposed models deliver higher relevance with dialogue history than baselines.

Language Model Adaption for Reinforcement Learning with Natural Language Action Space (2024.acl-long)

Copied to clipboard

Jiangxing Wang, Jiachen Li, Xiao Han, Deheng Ye, Zongqing Lu

Few-shot learning through contextual data augmentation (2021.eacl-main)

Challenge:	Previous research has focused on reducing the size of the natural language action space due to the combinatorial nature of the language.
Approach:	They propose mutual-information regularized policy optimization to reduce the action space by dynamically adjusting the prior provided by the pretrained model.
Outcome:	The proposed method improves monotonically on the mutual-information regularized RL objective.

Copied to clipboard

Farid Arthaud, Rachel Bawden, Alexandra Birch

Leveraging Only the Category Name for Aspect Detection through Prompt-based Constrained Clustering (2022.findings-emnlp)

Challenge:	Various strategies have been explored to learn from a journalist's post-edits . state-of-the-art APE systems require large numbers of post- edits for training .
Approach:	They propose to teach a pre-trained machine translation model to translate previously unseen words accurately . they extend a data augmentation approach to create training examples with similar contexts .
Outcome:	The proposed model improves accuracy on the scale of one to five examples with only 1 to 5 examples.

Copied to clipboard

Yazheng Li, Pengyun Wang, Yasheng Wang, Yong Dai, Yadao Wang, Lujia Pan, Zenglin Xu

Multi-Stage Prompting for Knowledgeable Dialogue Generation (2022.findings-acl)

Challenge:	Aspect category detection (ACD) aims to automatically identify user-concerned aspects from online reviews.
Approach:	They propose a method that relies on the category name of each aspect and a pretrained language model to generate constraints for clustering.
Outcome:	The proposed framework performs better than existing weakly supervised methods on nine benchmark datasets.

Copied to clipboard

Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, Bryan Catanzaro

Knowledge Graph Generation From Text (2022.findings-emnlp)

Challenge:	Existing knowledge-grounded dialogue systems typically use finetuned versions of a pretrained language model and large-scale knowledge bases.
Approach:	They propose a multi-stage prompting approach to generate knowledgeable responses from a single pretrained LM.
Outcome:	The proposed model outperforms the state-of-the-art retrieval-based model in terms of knowledge relevance and correctness by 5.8% and 5%, respectively.

Copied to clipboard

Igor Melnyk, Pierre Dognin, Payel Das

Structured Data in NLP

Challenge:	Existing methods for generating text from text are limited due to non-unique graph representation, complex node structure, large output spaces and limited parallel training data.
Approach:	They propose a novel end-to-end multi-stage Knowledge Graph generation system from textual inputs that separates the overall process into two stages.
Outcome:	The proposed system outperforms existing methods on a WebNLG 2020 Challenge dataset and on TekGen datasets.

An Unsupervised Framework for Adaptive Context-aware Simplified-Traditional Chinese Character Conversion (2024.lrec-main)

Copied to clipboard

Wei Li, Shutan Huang, Yanqiu Shao

Hexatagging: Projective Dependency Parsing as Tagging (2023.acl-short)

Challenge:	Traditional Chinese characters are still widely used in many areas of China . traditional methods to convert between simplified characters are ineffective .
Approach:	They propose an unsupervised adaptive context-aware conversion model that learns to convert between simplified and traditional Chinese characters under a denoising auto-encoder framework.
Outcome:	The proposed model outperforms strong unsupervised baselines and yields better conversion result for one-to-many cases.

Copied to clipboard

Afra Amini, Tianyu Liu, Ryan Cotterell

Unified Neural Topic Model via Contrastive Learning and Term Weighting (2023.eacl-main)

Challenge:	Using a pretrained language model, we can train language models on increasingly large amounts of data.
Approach:	They propose a dependency parser that constructs dependency trees by tagging words with elements from a finite set of possible tags.
Outcome:	The proposed approach achieves state-of-the-art performance of 96.4 LAS and 97.4 UAS on the Penn Treebank test set.

Copied to clipboard

Sungwon Han, Mingi Shin, Sungkyu Park, Changwook Jung, Meeyoung Cha

Information Extraction & Text Mining

Challenge:	Recent techniques employ pretrained language models to improve topic quality.
Approach:	They propose a topic-based model that uses contrastive learning and term weighting to learn from a pretrained language model and discover influential terms from semantically coherent clusters.
Outcome:	The proposed model outperforms baselines across multiple topic coherence measures and can be used as an add-on to existing topic models and improves their performance.

You should evaluate your language model on marginal likelihood over tokenisations (2021.emnlp-main)

Copied to clipboard

Kris Cao, Laura Rimell

Sign Language Production With Avatar Layering: A Critical Use Case over Rare Words (2022.lrec-1)

Challenge:	Neural language models typically tokenise input text into sub-word units to achieve an open vocabulary.
Approach:	They propose that language models should be evaluated on their marginal likelihood over tokenisations instead.
Outcome:	The proposed approach is unsatisfactory and may bottleneck model out-of-domain performance.

Copied to clipboard

Jung-Ho Kim, Eui Jun Hwang, Sukmin Cho, Du Hui Lee, Jong Park

Machine Translation

Challenge:	Existing vision-based sign language production approaches suffer from out-of-vocabulary (OOV) and test-time generalization problems.
Approach:	They propose an avatar-based sign language production system that generates sign language videos from spoken language expressions.
Outcome:	The proposed system achieves higher BLEU-4 and higher ROUGE-L scores on a new Korean-Korean sign language dataset.

On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation (2021.acl-long)

Copied to clipboard

Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, Luo Si

In-Context Learning for Few-Shot Dialogue State Tracking (2022.findings-emnlp)

Challenge:	Existing studies have shown that adapter-based tuning is more parameter-efficient than fine-tuning.
Approach:	They propose to add adapter modules to a pretrained language model and update the parameters of adapter module when learning on a downstream task.
Outcome:	The proposed method outperforms fine-tuning on low-resource and cross-lingual tasks and settings.

Copied to clipboard

Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, Mari Ostendorf

Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies (2023.emnlp-main)

Challenge:	Existing methods for zero-shot and few-shot learning dialogue state tracking are hard and expensive.
Approach:	They propose an in-context learning framework for zero-shot and few-shot learning dialogue state tracking (DST) a large pretrained language model takes a test instance and a few exemplars as input and directly decodes the dialogue state .
Outcome:	The proposed framework outperforms state-of-the-art models in few-shot settings . it is flexible and scalable, and requires less data to adapt to new domains and scenarios .

Copied to clipboard

Zhengxuan Wu, Alex Tamkin, Isabel Papadimitriou

MetaICL: Learning to Learn In Context (2022.naacl-main)

Challenge:	Large language models (LLMs) require vast datasets for pretraining, making it difficult to train LLMs from scratch for lowresource languages.
Approach:	They propose to transform a language of the GLUE benchmark and then fine tune a pretrained model on that dataset.
Outcome:	The proposed models recover from syntactic-style shifts, but cannot recover from vocabulary misalignment and embedding matrix re-initialization, even with continued pretraining on 15 million tokens.

Copied to clipboard

Sewon Min, Mike Lewis, Luke Zettlemoyer, Hannaneh Hajishirzi

Improving Abstraction in Text Summarization (D18-1)

Challenge:	Large language models can do in-context learning by conditioning on a few training examples with no parameter updates or task-specific templates.
Approach:	They propose a meta-training framework where a pretrained language model is tuned to do in-context learning on a large set of training tasks.
Outcome:	The proposed framework outperforms baseline models on 142 NLP datasets and a range of target tasks with domain shifts.

Copied to clipboard

Wojciech Kryściński, Romain Paulus, Caiming Xiong, Richard Socher

Don’t Miss the Labels: Label-semantic Augmented Meta-Learner for Few-Shot Text Classification (2021.findings-acl)

Challenge:	Abstractive text summarization models do not capture the abstractive nature of high quality summaries.
Approach:	They propose to decompose a decoder into a contextual network and a pretrained language model that incorporates prior knowledge about language generation.
Outcome:	The proposed model achieves comparable results to state-of-the-art models, based on ROUGE scores and human evaluations, while producing a significantly higher level of abstraction.

Copied to clipboard

Qiaoyang Luo, Lingqiao Liu, Yuhao Lin, Wei Zhang

Improving Controllable Text Generation with Position-Aware Weighted Decoding (2022.findings-acl)

Challenge:	Existing studies focus on building a meta-learner from input text but ignore abundant semantic information beneath class labels.
Approach:	They propose a framework to make full use of label semantics in few-shot text classification systems.
Outcome:	The proposed framework can be plugged into the existing few-shot text classification system.

Copied to clipboard

Yuxuan Gu, Xiaocheng Feng, Sicheng Ma, Jiaming Wu, Heng Gong, Bing Qin

Toward Human-Like Evaluation for Natural Language Generation with Error Analysis (2023.acl-long)

Challenge:	Controllable text generation is a challenging task in natural language generation, which aims to generate diverse text related to specified attributes.
Approach:	They propose a framework that uses a lightweight controller to adjust bias signals from the controller at different decoding positions.
Outcome:	Experiments on positive sentiment control, topic control, and language detoxification show the proposed framework works on 4 SOTA models.

Copied to clipboard

Qingyu Lu, Liang Ding, Liping Xie, Kanjian Zhang, Derek F. Wong, Dacheng Tao

Rethinking Style Transformer with Energy-based Interpretation: Adversarial Unsupervised Style Transfer using a Pretrained Model (2022.emnlp-main)

Challenge:	Pre-trained language models (PLMs) have been used to evaluate language generation tasks . pretrained error analysis can be used to refine the generated sentence toward higher confidence .
Approach:	They propose to combine pretrained language model based metrics with human-like error analysis to improve sentence confidence.
Outcome:	The proposed method outperforms top-scoring metrics in 19/25 settings.

Copied to clipboard

Hojun Cho, Dohee Kim, Seungwoo Ryu, ChaeHun Park, Hyungjong Noh, Jeong-in Hwang, Minseok Choi, Edward Choi, Jaegul Choo

SciBERT: A Pretrained Language Model for Scientific Text (D19-1)

Challenge:	Existing methods to train text style transfer models with adversarial loss degrade fluency compared to other metrics.
Approach:	They propose a method which leverages a pretrained language model to improve fluency by restructuring the discriminator and the model itself.
Outcome:	The proposed model achieves state-of-the-art on three public benchmarks and achieved state-outperformance on the overall metrics.

Copied to clipboard

Iz Beltagy, Kyle Lo, Arman Cohan

Prompt-Based Editing for Text Style Transfer (2023.findings-emnlp)

Challenge:	SciBERT is a pretrained language model based on BERT to improve performance on scientific NLP tasks.
Approach:	They propose a pretrained language model based on BERT to improve NLP performance . they evaluate on sequence tagging, sentence classification and dependency parsing .
Outcome:	The proposed model improves on sequence tagging, sentence classification and dependency parsing tasks with datasets from a variety of scientific domains.

Copied to clipboard

Guoqing Luo, Yu Han, Lili Mou, Mauajama Firdaus

AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain (2021.emnlp-main)

Challenge:	Text style transfer is a type of textual prompt that generates style-transferred texts word by word . early prediction errors may affect future word predictions.
Approach:	They propose a prompt-based editing approach to text style transfer using a pretrained language model.
Outcome:	The proposed approach outperforms existing systems with 20 times more parameters on three style-transfer benchmark datasets.

Copied to clipboard

Jimin Hong, TaeHee Kim, Hyesu Lim, Jaegul Choo

Automatic Multi-Label Prompting: Simple and Interpretable Few-Shot Classification (2022.naacl-main)

Challenge:	Existing methods to fine-tune a language model with a large corpus in a general domain are suboptimal for downstream data when domain discrepancy exists.
Approach:	They propose to consider the pretrained vocabulary as an optimizable parameter . they add domain specific vocabulary based on a tokenization statistic . their method achieved consistent performance improvements on diverse domains .
Outcome:	The proposed method achieves consistent performance improvements on diverse domains.

Copied to clipboard

Han Wang, Canwen Xu, Julian McAuley

Toward Interpretable Semantic Textual Similarity via Optimal Transport-based Contrastive Sentence Learning (2022.acl-long)

Challenge:	Prompt-based learning is an emerging paradigm for exploiting knowledge learned by a pretrained language model.
Approach:	They propose a method to automatically select label mappings for few-shot text classification with prompting.
Outcome:	The proposed method achieves competitive performance on the GLUE benchmark without human effort or external resources.

Copied to clipboard

Seonghyeon Lee, Dongha Lee, Seongbo Jang, Hwanjo Yu

Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering (2020.acl-main)

Challenge:	Existing studies have shown that a pretrained language model can capture sentence similarity but there is no interpretation method for the sentence similarities.
Approach:	They propose a pretrained language model that captures sentence similarity between embeddings and a transport-based distance measure that leverages semantically-aligned token pairs.
Outcome:	The proposed framework outperforms baselines on both STS and interpretable-STS benchmarks and provides interpretation consistent with human judgement.

Copied to clipboard

Alexander Fabbri, Patrick Ng, Zhiguo Wang, Ramesh Nallapati, Bing Xiang

AESOP: Paraphrase Generation with Adaptive Syntactic Control (2021.emnlp-main)

Challenge:	Question Answering (QA) is a field of increasing demand due to the availability of information online.
Approach:	They propose an unsupervised approach to training QA models with generated pseudo-training data by applying a simple template on a related sentence rather than the original context sentence.
Outcome:	The proposed approach improves the performance of a QA model on generated pseudo-training data.

Copied to clipboard

Jiao Sun, Xuezhe Ma, Nanyun Peng

Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning (2022.findings-emnlp)

Challenge:	Existing models for paraphrase generation use fixed syntactic structures for all input sentences.
Approach:	They propose to add syntactical control to a pretrained language model to generate fluent paraphrases using a retrieval-based selection module.
Outcome:	The proposed model achieves state-of-the-art on semantic preservation and syntactic conformation on two benchmark datasets with ground-truth syntaktic control from human-annotated exemplars.

Copied to clipboard

Shuo Xie, Jiahao Qiu, Ankita Pasad, Li Du, Qing Qu, Hongyuan Mei

A Systematic Study of Performance Disparities in Multilingual Task-Oriented Dialogue Systems (2023.emnlp-main)

Challenge:	Existing approaches to transfer a pretrained language model include fine-tuning all the parameters in the language model and adapting all its subsets.
Approach:	They propose to select layers based on the variability of their hidden states given a task-specific corpus.
Outcome:	The proposed model reduces the computational cost of transfer learning methods without sacrificing performance.

Copied to clipboard

Songbo Hu, Han Zhou, Moy Yuan, Milan Gritta, Guchun Zhang, Ignacio Iacobacci, Anna Korhonen, Ivan Vulić

Natural Language Interfaces

Challenge:	Existing systems trained for Arabic or Turkish using annotated data fully parallel to English ToD data still exhibit diminished ToD task performance.
Approach:	They define new quantitative measures of absolute and relative equivalence in system performance, capturing disparities across languages and within individual languages.
Outcome:	The proposed measures capture disparities across languages and within individual languages.

An Empirical Study of Contextual Data Augmentation for Japanese Zero Anaphora Resolution (2020.coling-main)

Copied to clipboard

Ryuto Konno, Yuichiroh Matsubayashi, Shun Kiyono, Hiroki Ouchi, Ryo Takahashi, Kentaro Inui

Information Extraction & Text Mining

Challenge:	Existing methods to augment labeled data are limited by the scarcity of labeles . a method called contextual data augmentation (CDA) can be used to augment labels .
Approach:	They propose a data augmentation method that generates labeled training instances using a pretrained language model.
Outcome:	The proposed method can improve the quality of augmented training data compared to the conventional method.

Multi-Stage Pre-training for Low-Resource Domain Adaptation (2020.emnlp-main)

Copied to clipboard

Rong Zhang, Revanth Gangi Reddy, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avi Sil, Todd Ward

Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters (2022.emnlp-main)

Challenge:	Existing approaches to transfer learning target data to in-domain text . prior work has adapted pre-trained LMs to specific domains .
Approach:	They extend the vocabulary of a pretrained language model with domain-specific terms to create synthetic tasks that help it transfer to downstream tasks.
Outcome:	The proposed approaches show significant performance gains on extractive reading comprehension, document ranking and duplicate question detection tasks.

Copied to clipboard

Hongyu Zhao, Hao Tan, Hongyuan Mei

Green & Sustainable NLP

Challenge:	Adapter-tuning is a paradigm that transfers a pretrained language model to downstream tasks . Previously proposed adapters are all feed-forward neural networks .
Approach:	They propose to use tiny-attention attention with extremely small per-head dimensionality as adapters to modify hidden states at each position . they propose to average multiple attention heads' weights during deployment to reduce its inference computation cost.
Outcome:	The proposed adapter outperforms other adapter-tuning methods on the GLUE benchmark . it uses attention with extremely small per-head dimensionality to modify hidden states .

Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification (2020.coling-main)

Copied to clipboard

Timo Schick, Helmut Schmid, Hinrich Schütze

DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts (2021.acl-long)

Challenge:	Existing approaches to few-shot text classification require domain expertise and an understanding of the language model's abilities to define the mapping between words and labels.
Approach:	They propose a method that converts textual inputs to cloze questions that contain some form of task description and processes them with a pretrained language model to map the predicted words to labels.
Outcome:	The proposed approach performs almost as well as hand-crafted label-to-word mappings for a number of tasks with small amounts of training data.

Copied to clipboard

Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, Yejin Choi

Enhancing Few-Shot Topic Classification with Verbalizers. a Study on Automatic Verbalizer and Ensemble Methods (2024.lrec-main)

Challenge:	Decoding-time Experts is a decoding- time method for controlled text generation . it combines a pretrained language model with "expert" LMs and/or "anti-expert" experts .
Approach:	They propose a decoding-time method that combines a pretrained language model with "expert" LMs and/or "anti-expert" experts to generate controlled text.
Outcome:	The proposed method outperforms existing controllable generation methods on automatic and human evaluations.

Copied to clipboard

Quang Anh Nguyen, Nadi Tomeh, Mustapha Lebbah, Thierry Charnois, Hanene Azzag, Santiago Cordoba Muñoz

Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task (2020.emnlp-main)

Challenge:	Pretrained language models are increasingly being used for many tasks.
Approach:	They propose to use verbalizers to help interpret masked word distributions into output predictions.
Outcome:	The proposed approach outperforms models trained with individual templates while using significantly less resources.

Copied to clipboard

Dongyeop Kang, Eduard Hovy

Dynamic Contextualized Word Embeddings (2021.acl-long)

Challenge:	despite the success of contextualized language models, language models cannot capture textual coherence of a long, multi-sentence document.
Approach:	They propose a paragraph completion task that predicts masked sentences in a sentence . they propose SSPlanner that predict what to say first and guides the pretrained model .
Outcome:	The proposed model outperforms baseline generation models on the paragraph completion task in automatic and human evaluation.

Copied to clipboard

Valentin Hofmann, Janet Pierrehumbert, Hinrich Schütze

Pretrained Language Model Embryology: The Birth of ALBERT (2020.emnlp-main)

Challenge:	Static word embeddings that represent words by a single vector cannot capture word meaning in different linguistic and extralinguistic contexts.
Approach:	They propose dynamic contextualized word embeddings that represent words as a function of linguistic and extralinguistic contexts.
Outcome:	The proposed model models time and social space jointly, making them attractive for NLP tasks involving semantic variability.

Copied to clipboard

Cheng-Han Chiang, Sung-Feng Huang, Hung-yi Lee

Continuous Entailment Patterns for Lexical Inference in Context (2021.emnlp-main)

Challenge:	Pretraining of pretrained models (LMs) has been extensively studied, but what happened during pretraining is rarely studied.
Approach:	They propose to use a totipotent language model to study pretraining behavior . they find that linguistic knowledge and world knowledge do not generally improve as pretraining proceeds, nor do downstream tasks’ performance.
Outcome:	The model learns to reconstruct and predict tokens of different parts of speech (POS) in different learning speeds during pretraining.

Copied to clipboard

Martin Schmitt, Hinrich Schütze

Towards Table-to-Text Generation with Pretrained Language Model: A Table Structure Understanding and Text Deliberating Approach (2022.emnlp-main)

Challenge:	Pretrained language models can be used to perform lexical inference in context tasks with relatively small training data.
Approach:	They propose to combine a pretrained language model with textual patterns to improve performance in both zero-shot and few-shot settings.
Outcome:	The proposed method compares pre-trained models with textual patterns on two established benchmarks for lexical inference in context (LIiC) the results show that the proposed patterns improve performance on LIiC, setting a new state of the art.

Copied to clipboard

Miao Chen, Xinjiang Lu, Tong Xu, Yanyan Li, Zhou Jingbo, Dejing Dou, Hui Xiong

Incremental Neural Lexical Coherence Modeling (2020.coling-main)

Challenge:	Currently, the generalization issues hinder the applicability of neural table-to-text models due to the limited source tables.
Approach:	They propose a table-structureaware text generation model with pretrained language model and propose TASD to bridge the gap between the structured table and text input.
Outcome:	The proposed model bridges the gap between the structured table and text input and generates accurate and fluent descriptive texts on two public datasets.

Copied to clipboard

Sungho Jeon, Michael Strube

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues (2023.acl-long)

Challenge:	Recent work on pretrained language models has led to significant improvements in a range of NLP tasks.
Approach:	They propose a coherence model which interprets sentences incrementally to capture lexical relations between them.
Outcome:	The proposed model interprets sentences incrementally to capture lexical relations between them.

Copied to clipboard

Yunxin Li, Baotian Hu, Chen Xinyu, Yuxin Ding, Lin Ma, Min Zhang

Multimodality

Challenge:	Existing methods for conditional inference on joint textual and visual clues lack multimodal context reasoning capability.
Approach:	They propose a multi-modal context reasoning approach that embeds textual semantics and objective image information into the pretrained language model to perform context reasoning.
Outcome:	The proposed approach improves on two data sets and shows 4.8% gain on the PMR.

Centering-based Neural Coherence Modeling with Hierarchical Discourse Segments (2020.emnlp-main)

Copied to clipboard

Sungho Jeon, Michael Strube

Linguistics & Cognitive NLP

Challenge:	Prior studies of coherence focused on identifying semantic relations between adjacent sentences.
Approach:	They propose a coherence model which takes discourse structural information into account without relying on human annotations.
Outcome:	The proposed model performs state-of-the-art on automated essay scoring and assessing writing quality tasks.

LM-Critic: Language Models for Unsupervised Grammatical Error Correction (2021.emnlp-main)

Copied to clipboard

Michihiro Yasunaga, Jure Leskovec, Percy Liang

Learning Cross-Task Dependencies for Joint Extraction of Entities, Events, Event Arguments, and Relations (2022.emnlp-main)

Challenge:	Recent work casts GEC as a translation problem using encoder-decoder models to map bad (ungrammatical) sentences into good (grammatically) sentences.
Approach:	They propose to use a pretrained language model to define an LM-Critic that judges a sentence to be grammatical if the LM assigns it a higher probability than its local perturbations.
Outcome:	The proposed method outperforms existing methods in both the unsupervised and supervised setting.

Copied to clipboard

Minh Van Nguyen, Bonan Min, Franck Dernoncourt, Thien Nguyen

Information Extraction & Text Mining

Challenge:	Existing work on IE tasks that use two types of dependencies is not optimal . emr, event trigger detection, event argument extraction, and relation extraction are challenging .
Approach:	They propose a model that learns cross-task dependencies from data . they treat each task instance as a node in a dependency graph .
Outcome:	The proposed model outperforms strong baselines over four datasets with different languages.

Recall, Expand, and Multi-Candidate Cross-Encode: Fast and Accurate Ultra-Fine Entity Typing (2023.acl-long)

Copied to clipboard

Chengyue Jiang, Wenyang Hui, Yong Jiang, Xiaobin Wang, Pengjun Xie, Kewei Tu

Unsupervised Multi-View Post-OCR Error Correction With Language Models (2021.emnlp-main)

Challenge:	State-of-the-art (SOTA) methods use the cross-encoder architecture to concatenate a mention (and its context) with each type and feed it into a pretrained language model (PLM) to score their relevance.
Approach:	They propose to perform entity typing in a recall-expand-filter manner and use a novel model to encode and score all these K candidates in one forward pass.
Outcome:	The proposed method is thousands of times faster than the CE-based architecture and is very efficient in fine-grained (130 types) and coarse-grain (9 types) entity typing.

Copied to clipboard

Harsh Gupta, Luciano Del Corro, Samuel Broscheit, Johannes Hoffart, Eliot Brenner

HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks (2023.findings-acl)

Challenge:	Prior work used text generation techniques or redundancy in similar passages for OCR error correction, which is not appropriate in cases of low corpus redundancies or weak document contextual information.
Approach:	They propose to use a pretrained language model to reconcile different OCR views in unsupervised way so that their combination contains fewer errors than each individual view.
Outcome:	The proposed model can reconcile multiple OCR views so that their combined version contains fewer errors than the best OCR view.

Copied to clipboard

Zhengkun Zhang, Wenya Guo, Xiaojun Meng, Yasheng Wang, Yadao Wang, Xin Jiang, Qun Liu, Zhenglu Yang

Exploring and Verbalizing Academic Ideas by Concept Co-occurrence (2023.acl-long)

Challenge:	Pretraining and fine-tuning are the dominant paradigms in natural language processing.
Approach:	They propose a parameter-efficient multitask learning framework that takes trainable hyper-embeddings and visual modality as input and outputs weights for different modules in a pretrained language model.
Outcome:	The proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods.

Copied to clipboard

Yi Xu, Shuqian Sheng, Bo Xue, Luoyi Fu, Xinbing Wang, Chenghu Zhou

Python Code Generation by Asking Clarification Questions (2023.acl-long)

Challenge:	a new framework for academic idea inspiration is being developed for academic research assistants . number of academic publications is increasing exponentially, making it difficult for an independent researcher to understand these papers thoroughly.
Approach:	They propose a framework based on concept co-occurrence for academic idea inspiration . they construct evolving concept graphs according to the co-existence relationship of concepts from 20 disciplines or topics .
Outcome:	The proposed system can be used to explore connections between academic concepts and verbalize the new ideas.

Copied to clipboard

Haau-Sing (Xiaocheng) Li, Mohsen Mesgar, André Martins, Iryna Gurevych

One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks (2024.findings-emnlp)

Challenge:	Recent work addresses text-to-code generation using pretrained language models (PLMs) for large-scale NLD: Logistic Regression.
Approach:	They propose a dataset containing pairs of natural language descriptions and code with created synthetic clarification questions and answers to solve the under-specified nature of a natural language description.
Outcome:	The proposed model improves on previous models, while introducing new challenges to the community, including when and what clarification questions should be asked.

Copied to clipboard

Sebastian Nehrdich, Oliver Hellwig, Kurt Keutzer

Adversarial Knowledge Stimulated Contrastive Prompting for Few-shot Language Learners (2023.findings-acl)

Challenge:	Morphologically rich languages are notoriously challenging to process for downstream NLP applications.
Approach:	They propose a pretrained model for NLP applications involving the morphologically rich language Sanskrit that outperforms previous models by a considerable margin.
Outcome:	The proposed model outperforms tokenized models on established Sanskrit word segmentation tasks and matches the current best lexicon-based model.

Copied to clipboard

Kai Zheng, Qingfeng Sun, Yaming Yang, Tengchao Lv, Yeyong Pi, Changlin Zhao, Fei Xu, Qi Zhang

Robustness of Learning from Task Instructions (2023.findings-acl)

Challenge:	Prompt-based fine-tuning has boosted performance of Pre-trained language models on few-shot Natural Language Understanding (NLU) tasks by employing task-specific prompts.
Approach:	They propose a Cloze-driven prompt framework for prompt tuning that implicitly stimulates knowledge from pre-trained language models.
Outcome:	The proposed framework outperforms state-of-the-art for prompt-based fine-tuning on few-shot NLU tasks.

Copied to clipboard

Jiasheng Gu, Hongyu Zhao, Hanzi Xu, Liangyu Nie, Hongyuan Mei, Wenpeng Yin

Robustness in NLP

Challenge:	traditional supervised learning mostly works on individual tasks and requires training on a large set of task-specific examples.
Approach:	a new study investigates the system robustness when instructions are manipulated and paraphrased . task instructions give the model the definition of the task and allow it to output the appropriate answer .
Outcome:	a new study shows that supervised learning is robust when instructions are manipulated, paraphrased or iii from different levels of conciseness.

Revisiting the Self-Consistency Challenges in Multi-Choice Question Formats for Large Language Model Evaluation (2024.lrec-main)

Copied to clipboard

Wenjie Zhou, Qiang Wang, Mingzhou Xu, Ming Chen, Xiangyu Duan

GiLT: Augmenting Transformer Language Models with Dependency Graphs (2026.acl-long)

Challenge:	Multi-choice questions (MCQs) are a common method for assessing the world knowledge of large language models.
Approach:	They propose three knowledge-equivalent question variants to assess LLMs' world knowledge . they propose option position shuffle, option label replacement, and conversion to a True/False format .
Outcome:	The proposed questions are shuffle, label replacement, and True/False format.

Copied to clipboard

Tianyu Huang, Yida Zhao, Chuyan Zhou, Kewei Tu